Repository navigation
core: nemotron's checkpoint identity moves to core, and GLM's ranks compare the checkpoint by it - #900
core: nemotron's checkpoint identity moves to core, and GLM's ranks compare the checkpoint by it#900Bizuayeu wants to merge 1 commit into
Conversation
…re the checkpoint by it, and a hub snapshot's revision only against another
bce5c60 to
781595e
Compare
Bizuayeu
changed the title
|
Rebased onto main as |
|
Thanks @Bizuayeu for reusing the released checkpoint identity instead of adding another hash. That is the right common owner for Nemotron and GLM startup agreement. We have prepared the per-file content-digest cache and GLM integration so unchanged shards need not be streamed repeatedly for hashing. Changed files and corrupt cache records force revalidation; metadata or equal headers alone are not checkpoint identity. The host controls cover those cases. Final cross-rank startup, changed-checkpoint refusal and load-time measurements still need their device receipts. Your transport checks remain attached to their exact tested head. Posted by Ash's AI Assistant Synthena |
|
Thank you. One detail for the per-file digest cache. Both passes here take their file list from the index's weight map: the identity before loading, and the digest of every byte. So a shard in a subfolder counts like any other. NVIDIA's FP4-Local checkpoints depend on that (#1015):
Those checkpoints stay covered if the cache takes its files from the same weight-map list, rather than from a listing of |
…d per-file digest cache, for every family Nemotron's checkpoint identity moves to core/checkpoint_identity.zig, as #900 proposed, and its whole-file and sorted non-expert digests now come from one digest-cache entry a file: the first full read computes both, and a warm start of the same shards, whole or split across ranks, reads neither again. Split manifests and Hugging Face blob names are still checked at every finish. The Nemotron engine, its workers, Super VL and the CLI read the shared module; the fixtures, including a snapshot's blob-name check, move with it to checkpoint_identity_test.zig. Co-authored-by: Bizuayeu <30091157+Bizuayeu@users.noreply.github.com>
…igest cache, so a warm start reads no shard The ranks' fingerprint carried a SHA-256 that cuda_artifact streamed over the index and every shard at each start, in one thread: 164 GB, about six minutes for GLM-5.3-Flash EXL3 on a GB10, every time. Each rank now sends the shared checkpoint identity (core/checkpoint_identity.zig: the metadata, then every shard byte read once in eight threads and kept per file in the digest cache), so a second start of unchanged shards reads none of them to compare; the fingerprint's layout is 9. A test writes three one-tensor checkpoints in folders of the same name: different tensor bytes under identical headers disagree, and a copy agrees. The comparison and its mismatch report follow #900's. Co-authored-by: Bizuayeu <30091157+Bizuayeu@users.noreply.github.com>
|
Thank you, @Bizuayeu, for moving checkpoint identity into the common core and exposing the rank-fingerprint gap. TensorFold 1.1.0 carries shared identity in 20bca34 and the GLM content comparison in 80d99ce, with your co-author credit, on the digest cache from 018e73c. Digests are reused only while the file's recorded identity is unchanged; corrupt or foreign indexes are ignored. Different tensor bytes under identical headers still disagree. Warm starts reuse unchanged shard digests while the ranks compare complete checkpoint content before loading. The final EXL3 GB10 pair passes 17/17 drafted/plain and 12/12 cached/cache-off chat checks. Those serving results do not replace the separate fingerprint-mismatch validation below. Host mismatch fixtures passed; two GB10s reused 83/83 shard digests on restart; a deliberately mismatched live pair is not recorded. Posted by Ash's AI Assistant Synthena |
A deliberately mismatched live pair on 1.1.0 (
Both ranks ended about 10 s after starting, before loading any weights. The first start on a cold cache read 190.4 GiB in 288.6 s (rank 0) and 295.8 s (rank 1), 0/33 cached; every later start in the same containers read 0.0 GiB, 33/33 cached. |
|
Shipped in 1.1.0 as Posted by Ash's AI Assistant Synthena |
…d per-file digest cache, for every family Nemotron's checkpoint identity moves to core/checkpoint_identity.zig, as ashhart#900 proposed, and its whole-file and sorted non-expert digests now come from one digest-cache entry a file: the first full read computes both, and a warm start of the same shards, whole or split across ranks, reads neither again. Split manifests and Hugging Face blob names are still checked at every finish. The Nemotron engine, its workers, Super VL and the CLI read the shared module; the fixtures, including a snapshot's blob-name check, move with it to checkpoint_identity_test.zig. Co-authored-by: Bizuayeu <30091157+Bizuayeu@users.noreply.github.com>
…igest cache, so a warm start reads no shard The ranks' fingerprint carried a SHA-256 that cuda_artifact streamed over the index and every shard at each start, in one thread: 164 GB, about six minutes for GLM-5.3-Flash EXL3 on a GB10, every time. Each rank now sends the shared checkpoint identity (core/checkpoint_identity.zig: the metadata, then every shard byte read once in eight threads and kept per file in the digest cache), so a second start of unchanged shards reads none of them to compare; the fingerprint's layout is 9. A test writes three one-tensor checkpoints in folders of the same name: different tensor bytes under identical headers disagree, and a copy agrees. The comparison and its mismatch report follow ashhart#900's. Co-authored-by: Bizuayeu <30091157+Bizuayeu@users.noreply.github.com>
2 participants
The third follow-up promised on #804 (comment),
as one commit on main:
781595e4. Since 1.0.12 Nemotron's ranks already agree on a checkpoint's identity before anyweight loads (
0089ac9c), so GLM now uses that instead of a second hash of its own.What this changes
GLM's ranks compared the checkpoint by its folder's last name, which refused identical copies kept under other names.
Nemotron's
cuda_identity.zigmoves unchanged tocore/checkpoint_identity.zig(Nemotron's three callers and thenemotron.identityexport only change their import), and GLM's fingerprint carries itsidentity(): config.json,the index, and each shard's name, size and header in name order, the tensors' bytes unread. A hub snapshot's revision
(the folder under
snapshots/) is still compared, but only when both ranks read a hub snapshot. The fingerprint'slayout goes to 6.
GLM does not take
Content, the every-byte digest Nemotron's ranks compare at ready: GLM's ranks agree once, beforethe weights load, and that digest would need a second agreement after it.
Receipt
46a08c18(TensorFold 1.0.12) plus this change and the split start's probe (glm: a split start checks that every pair of ranks can exchange, before the weights load #899) on onebranch; three DGX Spark (GB10, sm_121, CUDA 13.3) over RoCE; nvidia/GLM-5.3-Flash-NVFP4 at revision
423acf37.pass, and at TP=2 four of the server's replies, compared by hash, rounds and accepted drafts, equal main's.
zig build test-cuda-host853 passed (852 at main),zig build test621 passed and 7skipped, with one new test beside the moved ones (the identity unchanged by a dense byte in another folder, and
changed by another tensor name),
zig fmt --checkclean,glm-refand the thin and synthetic lanes checks; at TP=3the concurrent (41 PASS) and shared checks, at TP=2 the concurrent, shared and cover checks; 0 failed.
lean_check:516, as main.
snapshot; the host tests cover the identity); Nemotron on GPUs (only its import changes, and its host tests ran).
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (the lanes checks above).